{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "\u003ca href=\"https://colab.research.google.com/github/tensorflow/tpu/blob/0e3cfbdfbcf321681c1ac1c387baf7a1a41d8d21/tools/colab/regression_sine_data_with_keras.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "N6ZDpd9XzFeN"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "KUu4vOt5zI9d"
      },
      "outputs": [],
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "afRHQRnkeSeJ"
      },
      "source": [
        "# A simple regression model using Keras with Cloud TPUs\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jKfvNSR-djeV"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demonstrates using Cloud TPUs in colab to build a simple regression model using **y = sin(x)** to predict y for given x.\n",
        "\n",
        "This model generates huge amounts of data that and demonstrates the training performance advantage of Cloud TPU over CPU.\n",
        "\n",
        "The model trains for 10 epochs with 512 steps per epoc on both TPU and CPU and completes in approximately 10 minutes.\n",
        "\n",
        "This notebook is hosted on GitHub. To view it in its original repository, after opening the notebook, select **File \u003e View on GitHub**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dgAHfQtuhddd"
      },
      "source": [
        "## Learning objectives\n",
        "\n",
        "In this Colab, you will learn how to:\n",
        "*   Create and compile a Keras model on TPU with a distribution strategy.\n",
        "*   Train, evaluate, and and generate predictions on Cloud TPU.\n",
        "*   Compare the performance of a TPU versus a CPU.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6b8OVJWtPID9"
      },
      "source": [
        "## Instructions\n",
        "\n",
        "\u003ch3\u003e  \u0026nbsp;\u0026nbsp;Train on TPU\u0026nbsp;\u0026nbsp; \u003ca href=\"https://cloud.google.com/tpu/\"\u003e\u003cimg valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/tpu-hexagon.png\" width=\"50\"\u003e\u003c/a\u003e\u003c/h3\u003e\n",
        "\n",
        "   1. On the main menu, click Runtime and select **Change runtime type**. Set \"TPU\" as the hardware accelerator.\n",
        "   1. Click Runtime again and select **Runtime \u003e Run All**. You can also run the cells manually with Shift-ENTER."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6IhP5cGzewbQ"
      },
      "source": [
        "### Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "JeMli385le2A"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import tensorflow.keras as keras\n",
        "import math\n",
        "import os\n",
        "import pprint\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from sklearn import metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "U-fdEJFNfICv"
      },
      "outputs": [],
      "source": [
        "print(tf.__version__)\n",
        "import distutils\n",
        "if distutils.version.LooseVersion(tf.__version__) < '1.14':\n",
        "    raise Exception('This notebook is compatible with TensorFlow 1.14 or higher, for TensorFlow 1.13 or lower please use the previous version at https://github.com/tensorflow/tpu/blob/r1.13/regression_sine_data_with_keras.ipynb')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kK51TQYnfiXO"
      },
      "source": [
        "### Resolve TPU Address"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YWmHCZrufCO4"
      },
      "outputs": [],
      "source": [
        "use_tpu = True #@param {type:\"boolean\"}\n",
        "\n",
        "if use_tpu:\n",
        "    assert 'COLAB_TPU_ADDR' in os.environ, 'Missing TPU; did you request a TPU in Notebook Settings?'\n",
        "\n",
        "if 'COLAB_TPU_ADDR' in os.environ:\n",
        "  TF_MASTER = 'grpc://{}'.format(os.environ['COLAB_TPU_ADDR'])\n",
        "else:\n",
        "  TF_MASTER=''\n",
        "\n",
        "with tf.Session(TF_MASTER) as session:\n",
        "  print ('List of devices:')\n",
        "  pprint.pprint(session.list_devices())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Vy1qljh-vWxW"
      },
      "source": [
        "### Creating data for y = sin(x).\n",
        "Sine wave data is created using numpy. And to make it more difficult, random noice is added to the sine wave."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "89lavEaRsMiu"
      },
      "outputs": [],
      "source": [
        "data_size = 2**18\n",
        "\n",
        "x = np.linspace(0, 6, data_size, dtype=np.float32)\n",
        "np.random.shuffle(x)\n",
        "\n",
        "y = -20 * np.sin(x, dtype=np.float32) + 3 + np.random.normal(0, 1, (data_size,)).astype(np.float32)\n",
        "\n",
        "x = x.reshape(-1, 1)\n",
        "y = y.reshape(-1, 1)\n",
        "\n",
        "train_x, test_x = x[:data_size//2], x[data_size//2:]\n",
        "train_y, test_y = y[:data_size//2], y[data_size//2:]\n",
        "\n",
        "plt.plot(x, y, 'bo')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XPNNUZ7ygjGk"
      },
      "source": [
        "### Define model:\n",
        "Model will have an input layer where it takes in the x coordinate, two densely connected layers with 200 and 80 nodes, and an output layer where it returns the predicted y value."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "mzINcHJdln91"
      },
      "outputs": [],
      "source": [
        "def get_model():\n",
        "  return keras.Sequential([\n",
        "      keras.layers.Dense(1, input_shape=(1,)),\n",
        "      keras.layers.Dense(200, activation='sigmoid'),\n",
        "      keras.layers.Dense(80, activation='sigmoid'),\n",
        "      keras.layers.Dense(1)\n",
        "  ])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aKk5vUvalpfF"
      },
      "source": [
        "### Compiling the model with a distribution strategy\n",
        "To make the model usable by a TPU, we first must create and compile it using a distribution strategy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ggY7UortwcgK"
      },
      "outputs": [],
      "source": [
        "resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TF_MASTER)\n",
        "tf.contrib.distribute.initialize_tpu_system(resolver)\n",
        "strategy = tf.contrib.distribute.TPUStrategy(resolver)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "j-2PBN7smmCy"
      },
      "outputs": [],
      "source": [
        "with strategy.scope():\n",
        "  model = get_model()\n",
        "  model.compile(optimizer=tf.keras.optimizers.SGD(.01),\n",
        "                loss='mean_squared_error',\n",
        "                metrics=['mean_squared_error'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ze8cEk8cw3oL"
      },
      "source": [
        "###Training of the model on TPU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "nedmSZk4lsE1"
      },
      "outputs": [],
      "source": [
        "model.fit(train_x, train_y, epochs=10, steps_per_epoch=512)\n",
        "model.save_weights('/tmp/sine.h5', overwrite=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZFphWL2ExXjC"
      },
      "source": [
        "###Prediction\n",
        "For predictions, same model architecture is being used which is loaded with the earlier learned weights."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "vpz43puBo3vz"
      },
      "outputs": [],
      "source": [
        "prediction_model = get_model()\n",
        "prediction_model.load_weights('/tmp/sine.h5')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "1Vc44SqfluY1"
      },
      "outputs": [],
      "source": [
        "predictions = prediction_model.predict(test_x)\n",
        "plt.plot(test_x, predictions, 'ro')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "q8Sdh-1n4pqt"
      },
      "source": [
        "###Training on CPU for comparison with TPU\n",
        "How much time did we save by using a TPU? Let's try training the exact same model on the same data, but without the TPU speedup.\n",
        "\n",
        "**(WARNING: This will take approximately 7 minutes to execute!)**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YCkqpEXu4Xx4"
      },
      "outputs": [],
      "source": [
        "slow_model = get_model()\n",
        "slow_model.compile(optimizer=tf.train.GradientDescentOptimizer(.01),\n",
        "             loss='mean_squared_error',\n",
        "             metrics=['mean_squared_error'])\n",
        "slow_model.fit(train_x, train_y, epochs=10, steps_per_epoch=512)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2a5cGsSTEBQD"
      },
      "source": [
        "## What's next\n",
        "\n",
        "* Learn about [Cloud TPUs](https://cloud.google.com/tpu/docs) that Google designed and optimized specifically to speed up and scale up ML workloads for training and inference and to enable ML engineers and researchers to iterate more quickly.\n",
        "* Explore the range of [Cloud TPU tutorials and Colabs](https://cloud.google.com/tpu/docs/tutorials) to find other examples that can be used when implementing your ML project.\n",
        "\n",
        "On Google Cloud Platform, in addition to GPUs and TPUs available on pre-configured [deep learning VMs](https://cloud.google.com/deep-learning-vm/),  you will find [AutoML](https://cloud.google.com/automl/)*(beta)* for training custom models without writing code and [Cloud ML Engine](https://cloud.google.com/ml-engine/docs/) which will allows you to run parallel trainings and hyperparameter tuning of your custom models on powerful distributed hardware.\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "collapsed_sections": [
        "N6ZDpd9XzFeN"
      ],
      "name": "Simple Regression Model Using Keras on Colab TPU - [Full Sine as input]",
      "provenance": [],
      "version": "0.3.2"
    },
    "kernelspec": {
      "display_name": "Python 2",
      "name": "python2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
